{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"right\" style=\"max-width: 200px; height: auto\" src=\"cfds_logo.png\">\n",
    "\n",
    "###  Lab 11 - \"Autoencoder Neural Networks\"\n",
    "\n",
    "Chartered Financial Data Scientist (CFDS), Spring Term 2020"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, we will learn how to apply another type of deep learning technique referred to as **Autoencoder Neural Networks (AENN)**. \n",
    "\n",
    "Unlike standard feedforward neural networks, AENN's learn how to **encode** the input data into a low dimensional representation.  At the same time, the AENN learns how to **decode** the original data back from the encoded representation. The decoded data usually referred to as \"reconstruction\", should match the original input as closely as possible.\n",
    "\n",
    "We will again use the functionality of the **PyTorch** library to implement and train an AENN. The network will be trained to learn the characteristics of historical accounting data, usually referred to as \"journal entries.\" Once the model is trained, we will apply it to detect anomalous journal entries contained in the dataset. Finally, we will inspect the low-dimensional representations of each journal entry to interpret the detection results.\n",
    "\n",
    "The figure below illustrates a high-level view on the machine learning process we aim to establish in this lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"center\" style=\"max-width: 900px\" src=\"process.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lab Objectives:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After today's lab, you should be able to:\n",
    "\n",
    ">1. Understand the **basic concepts, intuitions and major building blocks** of autoencoder neural networks.\n",
    ">2. **Pre-process** categorical financial data to learn a model of its characteristics and pattern.\n",
    ">3. Apply autoencoder neural networks to **detect anomalies** in large-scale financial data.\n",
    ">4. **Interpret the detection results** of the network as well as its reconstruction loss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, pls. don't hesitate to ask all your questions either during the lab or send us an email via marco.schreyer@fds.ai or damian.borth@fds.ai."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start let's watch a motivational video:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "# GitHub Arctic Code Vault\n",
    "# YouTubeVideo('fzI9FNjXQ0o', width=800, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Association of Certified Fraud Examiners estimates in its Global Fraud Study 2016 [1] that the typical organization loses 5% of its annual revenues due to fraud. According to Joseph T. Wells [2] the term **\"fraud\"** refers to, **\"the abuse of one's occupation for personal enrichment through the deliberate misuse of an organization's resources or assets\"**. \n",
    "\n",
    "A similar more recent study, conducted by the auditors of PwC, revealed that 30% of the study respondents experienced losses of between USD 100,000 and USD 5 million [3] in the last 24 months. The study also showed that financial statement fraud caused by far the greatest median loss of the surveyed fraud schemes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification of Financial Anomalies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When conducting a detailed examination of real-world journal entries, usually recorded in large-scaled Accounting Information Systems (AIS) or Enterprise Ressource Planning (ERP) systems, two common characteristics can be observed:\n",
    "\n",
    "> - specific transactions attributes exhibit **a wide variety of distinct attribute values**, e.g., customer information, posted sub-ledgers, amount data, and \n",
    "> - the transactions exhibit **strong dependencies between specific attribute values** e.g., between customer information and type of payment, posting type, and general ledgers. \n",
    "\n",
    "Derived from this observation we distinguish two classes of anomalous journal entries, namely **\"global\"** and **\"local\" anomalies** as illustrated in **Figure 1** below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 600px; height: auto\" src=\"anomalies.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 1:** Illustrative example of global and local anomalies portrait in a feature space of the two transaction features \"Posting Amount\" (Feature 1) and \"Posting Positions\" (Feature 2)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Global Anomalies***, are financial transactions that exhibit **unusual or rare individual attribute values**. These anomalies usually relate to highly skewed attributes, e.g., seldom posting users, rarely used ledgers, or unusual posting times. Traditionally \"red-flag\" tests performed by auditors during annual audits are designed to capture those types of anomalies. However, such tests might result in a high volume of false-positive alerts due to, e.g., regular reverse postings, provisions, and year-end adjustments usually associated with a low fraud risk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Local Anomalies***, are financial transactions that exhibit an **unusual or rare combination of attribute values** while the individual attribute values occur quite frequently e.g. exceptional accounting records. This type of anomaly is significantly more challenging to detect since perpetrators intend to disguise their activities trying to imitate a normal behavior. As a result, such anomalies usually pose a high fraud risk since they might correspond to, e.g., misused user accounts, irregular combinations of general ledger accounts and posting keys that don't follow a usual activity pattern."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup of the Jupyter Notebook Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a next step, let's import the libraries needed throughout the lab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the previous labs, we need to import a couple of Python libraries that allow for data analysis and data visualization. We will mostly use the `PyTorch`, `Numpy`, `Sklearn`, `Matplotlib`, `Seaborn`, `BT`, and a few utility libraries throughout the lab:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import python data science and utility libraries\n",
    "import os, sys, itertools, urllib, io\n",
    "import datetime as dt\n",
    "import pandas as pd\n",
    "import pandas_datareader as dr\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the Python machine / deep learning libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pytorch libraries\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.utils import data\n",
    "from torch.utils.data import dataloader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import Python plotting libraries and set general plotting parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "plt.style.use('seaborn')\n",
    "plt.rcParams['figure.figsize'] = [10, 5]\n",
    "plt.rcParams['figure.dpi']= 150"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enable notebook matplotlib inline plotting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a notebook folder structure to store the data as well as the trained neural network models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('./data'): os.makedirs('./data')  # create data directory\n",
    "if not os.path.exists('./models'): os.makedirs('./models')  # create trained models directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set a random seed value to obtain reproducable results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init deterministic seed\n",
    "seed_value = 1234\n",
    "np.random.seed(seed_value) # set numpy seed\n",
    "torch.manual_seed(seed_value); # set pytorch seed CPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enable GPU computing by setting the `device` flag and init a `CUDA` seed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set cpu or gpu enabled device\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu').type\n",
    "\n",
    "# init deterministic GPU seed\n",
    "torch.cuda.manual_seed(seed_value)\n",
    "\n",
    "# log type of device enabled\n",
    "now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "print('[LOG {}] notebook with \\'{}\\' computation enabled'.format(str(now), str(device)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, let's display information about the potential GPUs running on the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nvidia-smi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's execute the cell below to display information about the `Python` and `PyTorch` version running on this notebook or compute server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print current Python version\n",
    "now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "print('[LOG {}] The Python version: {}'.format(now, sys.version))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print current PyTorch version\n",
    "now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "print('[LOG {}] The PyTorch version: {}'.format(now, torch.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Dataset Download and Data Assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nowadays, organizations accelerate the digitization and reconfiguration of business processes [4] affecting in particular Accounting Information Systems (AIS) or more general Enterprise Resource Planning (ERP) systems.\n",
    "\n",
    "Steadily, these systems collect vast quantities of electronic evidence at an almost atomic level. This observation holds in particular for the journal entries of an organization recorded in its general ledger and sub-ledger accounts. SAP, one of the most prominent ERP software providers, estimates that approx. 76% of the world's transaction revenue touches one of their systems [5].\n",
    "\n",
    "The illustration in **Figure 1** depicts a hierarchical view of an Accounting Information System (AIS) recording process and journal entry information in designated database tables. In the context of fraud examinations, the data collected by such systems may contain valuable traces of a potential fraud scheme."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 700px; height: auto\" src=\"accounting.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 1:** Hierarchical view of an Accounting Information System (AIS) that records distinct layers of abstraction, namely (1) the business process information, (2) the accounting information as well as the (3) technical journal entry information in designated database tables."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section of the lab notebook, we will conduct a descriptive analysis of the lab's financial dataset. Furthermore, we will apply some necessary pre-processing steps to train a deep neural network. The lab is based on a derivation of the **\"Synthetic Financial Dataset For Fraud Detection\"** by Lopez-Rojas [6] available via the Kaggle predictive modeling and analytics competitions platform that can be obtained using the following link: https://www.kaggle.com/ntnu-testimon/paysim1.\n",
    "\n",
    "Let's start loading the dataset and investigate its structure and attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset into the notebook kernel\n",
    "url = 'https://raw.githubusercontent.com/financial-data-science/CFDS/master/lab_11/data/fraud_dataset_v2.csv'\n",
    "ori_dataset = pd.read_csv(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect the datasets dimensionalities\n",
    "now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "print('[LOG {}] transactional dataset of {} rows and {} columns retreived.'.format(now, ori_dataset.shape[0], ori_dataset.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 Initial Data and Attribute Assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We augmented the dataset and renamed the attributes to mimic a real-world dataset that one usually observes in SAP-ERP systems as part of SAP's Finance and Cost controlling (FICO) module. \n",
    "\n",
    "The dataset contains a subset of in total seven categorical and two numerical attributes available in the FICO BKPF (containing the posted journal entry headers) and BSEG (containing the posted journal entry segments) tables. Please, find below a list of the individual attributes as well as a brief description of their respective semantics:\n",
    "\n",
    ">- `BELNR`: the accounting document number,\n",
    ">- `BUKRS`: the company code,\n",
    ">- `BSCHL`: the posting key,\n",
    ">- `HKONT`: the posted general ledger account,\n",
    ">- `PRCTR`: the posted profit center,\n",
    ">- `WAERS`: the currency key,\n",
    ">- `KTOSL`: the general ledger account key,\n",
    ">- `DMBTR`: the amount in the local currency,\n",
    ">- `WRBTR`: the amount in the document currency.\n",
    "\n",
    "Let's also have a closer look into the top 10 rows of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect top rows of dataset\n",
    "ori_dataset.head(10) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may also have noticed the attribute `label` in the data. We will use this field throughout the lab to evaluate the quality of our trained models. The field describes the true nature of each transaction of either being a **regular** transaction (denoted by `regular`) or an **anomaly** (denoted by `global` and `local`). Let's have a closer look into the distribution of the regular vs. anomalous transactions in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# number of anomalies vs. regular transactions\n",
    "ori_dataset.label.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, the statistic reveals that similar to real-world scenarios, we are facing a highly \"unbalanced\" dataset. Overall, the dataset contains only a small fraction of **100 (0.018%)** anomalous transactions. While the 100 anomalous entries encompass **70 (0.013%)** \"global\" anomalies and **30 (0.005%)** \"local\" anomalies as introduced in section 1.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove the \"ground-truth\" label information for the following steps of the lab\n",
    "label = ori_dataset.pop('label')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Pre-Processing of Categorical Transaction Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the initial data assessment above, we can observe that the majority of attributes recorded in AIS- and ERP-systems correspond to categorical (discrete) attribute values, e.g. the posting date, the general ledger account, the posting type, the currency. Let's have a more detailed look into the distribution of two dataset attributes, namely (1) the posting key `BSCHL` as well as (2) the general ledger account `HKONT`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare to plot posting key and general ledger account side by side\n",
    "fig, ax = plt.subplots(1, 2)\n",
    "fig.set_figwidth(20)\n",
    "\n",
    "# plot the distribution of the posting key attribute\n",
    "g = sns.countplot(x=ori_dataset['BSCHL'], ax=ax[0])\n",
    "\n",
    "# set axis labels\n",
    "g.set_xticklabels(g.get_xticklabels(), rotation=90)\n",
    "g.set_xlabel('BSCHL Value', fontsize=18)\n",
    "g.set_ylabel('Value Count', fontsize=18)\n",
    "\n",
    "# set plot title\n",
    "g.set_title('Distribution of the \\'Posting Key\\' attribute values', fontsize=20)\n",
    "\n",
    "# plot the distribution of the general ledger attribute\n",
    "g = sns.countplot(x=ori_dataset['HKONT'], ax=ax[1])\n",
    "\n",
    "# set axis labels\n",
    "g.set_xticklabels(g.get_xticklabels(), rotation=90)\n",
    "g.set_xlabel('HKONT Value', fontsize=18)\n",
    "g.set_ylabel('Value Count', fontsize=18)\n",
    "\n",
    "# set plot title\n",
    "g.set_title('Distribution of the \\'General Ledger\\' attribute values', fontsize=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, neural networks are, in general, not designed to be trained directly on categorical data and require the attributes to be trained on to be numeric. One simple way to meet this requirement is by applying a technique referred to as **\"one-hot\" encoding**. Using this encoding technique, we will derive a numerical representation of each of the categorical attribute values. One-hot encoding creates new binary columns for each categorical attribute value present in the original data. \n",
    "\n",
    "Let's have a look at the example shown in **Figure 2** below. The **categorical attribute “Receiver”** below contains the names \"John,\" \"Timur\" and \"Marco.\" We \"one-hot\" encode the names by creating a separate binary column for each possible name-value observable in the \"Receiver\" column. Now, we encode for each transaction that contains the value \"John\" in the \"Receiver\" column this observation with 1.0 in the newly created \"John\" column and 0.0 in all other generated name columns."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 600px; height: auto\" src=\"encoding.png\">\n",
    "\n",
    "**Figure 2:** Exemplary one-hot encoding of the distinct `Receiver` attribute values into specific binary (\"one-hot) columns. Thereby, each attribute value observable in the dataset results in a separate column. The column value `1.0` denotes the occurance of the attribute value in the corresponding journal entry. In contrast the column value `0.0` indicates the absence of the attribute value in the corresponding journal entry."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this technique will \"one-hot\" encode the six categorical attributes in the original transactional dataset. This can be achieved using the `get_dummies()` function available in the Pandas data science library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select categorical attributes to be \"one-hot\" encoded\n",
    "categorical_attr_names = ['KTOSL', 'PRCTR', 'BSCHL', 'HKONT']\n",
    "\n",
    "# encode categorical attributes into a binary one-hot encoded representation \n",
    "ori_dataset_cat_processed = pd.get_dummies(ori_dataset[categorical_attr_names])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's inspect the encoding of 10 sample transactions to see if the encoding was accomplished successfully;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect encoded sample transactions\n",
    "ori_dataset_cat_processed.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.3 Pre-Processing of Numerical Transaction Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now inspect the distributions of the two numerical attributes contained in the transactional dataset namely, the (1) local currency amount `DMBTR` and the (2) document currency amount `WRBTR`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the log-scaled \"DMBTR\" as well as the \"WRBTR\" attribute value distribution\n",
    "fig, ax = plt.subplots(1,2)\n",
    "fig.set_figwidth(20)\n",
    "\n",
    "# plot distribution of the local amount attribute\n",
    "g = sns.distplot(ori_dataset['DMBTR'].tolist(), ax=ax[0])\n",
    "\n",
    "# set axis labels\n",
    "g.set_xlabel('DMBTR Value', fontsize=18)\n",
    "g.set_ylabel('Value Count', fontsize=18)\n",
    "\n",
    "# set plot title\n",
    "g.set_title('Distribution of the \\'Local Amount\\' attribute values', fontsize=20)\n",
    "\n",
    "# plot distribution of the document amount attribute\n",
    "g = sns.distplot(ori_dataset['WRBTR'].tolist(), ax=ax[1])\n",
    "\n",
    "# set axis labels\n",
    "g.set_xlabel('WRBTR Value', fontsize=18)\n",
    "g.set_ylabel('Value Count', fontsize=18)\n",
    "\n",
    "# set plot title\n",
    "g.set_title('Distribution of the \\'Foreign Amount\\' attribute values', fontsize=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, it can be observed that for both attributes, the distributions of amount values are **heavy-tailed**. In order to approach faster a potential global minimum scaling and normalization of numerical input values is good practice. Therefore, we first log-scale both variables and second min-max normalize the scaled amounts to the interval [0, 1]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the 'DMBTR' and 'WRBTR' attribute\n",
    "numeric_attr_names = ['DMBTR', 'WRBTR']\n",
    "\n",
    "# add a small epsilon to eliminate zero values from data for log scaling\n",
    "numeric_attr = ori_dataset[numeric_attr_names] + 1e-7\n",
    "\n",
    "# log scale the 'DMBTR' and 'WRBTR' attribute values\n",
    "numeric_attr = numeric_attr.apply(np.log)\n",
    "\n",
    "# normalize all numeric attributes to the range [0,1]\n",
    "ori_dataset_num_processed = (numeric_attr - numeric_attr.min()) / (numeric_attr.max() - numeric_attr.min())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now visualize the log-scaled and min-max normalized distributions of both attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot the log-scaled \"DMBTR\" as well as the \"WRBTR\" attribute value distribution\n",
    "fig, ax = plt.subplots(1,2)\n",
    "fig.set_figwidth(20)\n",
    "\n",
    "# plot distribution of the local amount attribute\n",
    "g = sns.distplot(ori_dataset_num_processed['DMBTR'].tolist(), ax=ax[0])\n",
    "\n",
    "# set axis labels\n",
    "g.set_xlabel('DMBTR Value', fontsize=18)\n",
    "g.set_ylabel('Value Count', fontsize=18)\n",
    "\n",
    "# set plot title\n",
    "g.set_title('Distribution of the \\'Local Amount\\' attribute values', fontsize=20)\n",
    "\n",
    "# plot distribution of the document amount attribute\n",
    "g = sns.distplot(ori_dataset_num_processed['WRBTR'].tolist(), ax=ax[1])\n",
    "\n",
    "# set axis labels\n",
    "g.set_xlabel('WRBTR Value', fontsize=18)\n",
    "g.set_ylabel('Value Count', fontsize=18)\n",
    "\n",
    "# set plot title\n",
    "g.set_title('Distribution of the \\'Foreign Amount\\' attribute values', fontsize=20);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.4 Merge Categorical and Numerical Transaction Attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we merge both pre-processed numerical and categorical attributes into a single dataset that we will use for training our deep autoencoder neural network (explained an implemented in the following section 2):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# merge categorical and numeric subsets\n",
    "ori_subset_transformed = pd.concat([ori_dataset_cat_processed, ori_dataset_num_processed], axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's again have a look at the dimensionality of the dataset after we applied the distinct pre-processing steps to the attributes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect final dimensions of pre-processed transactional data\n",
    "ori_subset_transformed.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, upon completion of all the pre-processing steps (excl. the exercises), we should end up with an encoded dataset consisting of a total number of **533,009 records** (rows) and **384 encoded attributes** (columns). Let's keep the number number of columns in mind since it will define the dimensionality of the input- and output-layer of our deep autoencoder network, which we will now implement in the following section."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Autoencoder Neural Networks (AENNs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The objective of this section is to familiarise ourselves with the underlying idea and concepts of building a deep **Autoencoder Neural Network (AENN)**. We will cover the primary building blocks and the specific network structure of AENNs as well as an exemplary implementation using the open-source machine learning library `PyTorch`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Autoencoder Neural Network Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AENNs or \"Replicator Neural Networks\" is a variant of general feed-forward neural networks that have been initially introduced by Hinton and Salakhutdinov in [6]. AENNs usually comprise a **symmetrical network architecture** as well as a central hidden layer, referred to as **\"latent\"** or **\"coding\" layer**, of lower dimensionality. The design is chosen intentionally since the training objective of an AENN is to reconstruct its input in an \"unsupervised\" training setup. **Figure 3** below illustrates a schematic view of an autoencoder neural network:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"middle\" style=\"max-width: 800px; height: auto\" src=\"autoencoder.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 3:** Schematic view of an autoencoder network comprised of two non-linear mappings (fully connected feed forward neural networks) referred to as encoder $f_\\theta: \\mathbb{R}^{dx} \\mapsto \\mathbb{R}^{dz}$ and decoder $g_\\theta: \\mathbb{R}^{dz} \\mapsto \\mathbb{R}^{dx}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Furthermore, AENNs can be interpreted as \"lossy\" data **compression algorithms**. They are \"lossy\" in a sense that the reconstructed outputs will be degraded compared to the original inputs. The difference between the original input $x^i$ and its reconstruction $\\hat{x}^i$ is referred to as **reconstruction error**. In general, AENNs encompass three major building blocks:\n",
    "\n",
    "\n",
    ">   1. an encoding mapping function $f_\\theta$, \n",
    ">   2. a decoding mapping function $g_\\theta$, \n",
    ">   3. and a loss function $\\mathcal{L_{\\theta}}$.\n",
    "\n",
    "Most commonly the encoder and the decoder mapping functions consist of **several layers of neurons followed by a non-linear function** and shared parameters $\\theta$. The encoder mapping $f_\\theta(\\cdot)$ maps an input vector (e.g. an \"one-hot\" encoded transaction) $x^i$ to a compressed representation $z^i$ referred to as latent space $Z$. This hidden representation $z^i$ is then mapped back by the decoder $g_\\theta(\\cdot)$ to a re-constructed vector $\\hat{x}^i$ of the original input space (e.g. the re-constructed encoded transaction). Formally, the nonlinear mappings of the encoder- and the decoder-function can be defined by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>$f_\\theta(x^i) = s(Wx^i + b)$, and $g_\\theta(z^i) = s′(W′z^i + d)$,</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where $s$ and $s′$ denote non-linear activations with model parameters $\\theta = \\{W, b, W', d\\}$, $W \\in \\mathbb{R}^{d_x \\times d_z}, W' \\in \\mathbb{R}^{d_z \\times d_y}$ are weight matrices and $b \\in \\mathbb{R}^{dx}$, $d \\in \\mathbb{R}^{dz}$ are offset bias vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2 Autoencoder Neural Network Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's start implementing an `AENN model` by first implementing the encoder network using PyTorch. For the encoder, we aim to implement a network consisting of **nine fully-connected layers**. Furthermore, the encoder is specified by the following number of neurons per layer: \"384-256-128-64-32-16-8-4-3\". Meaning the first layer consists of 384 neurons (determined by the dimensionality of our input data), the second layer of 256 neurons and the subsequent layers of 128, 64, 32, 16, 8, 4 and 3 neurons respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some elements of the encoder network code below should be given particular attention:\n",
    "\n",
    ">- `self.encoder_Lx`: defines the linear transformation of the layer applied to the incoming input: $Wx + b$.\n",
    ">- `nn.init.xavier_uniform`: inits the layer weights using a uniform distribution according to [9]. \n",
    ">- `self.encoder_Rx`: defines the non-linear transformation of the layer: $\\sigma(\\cdot)$.\n",
    ">- `self.dropout`: randomly zeros some of the elements of the input tensor with probability $p$ according to [8].\n",
    "\n",
    "We use **\"Leaky ReLUs\"** as introduced by Xu et al. in [7] to avoid \"dying\" non-linearities and to speed up training convergence. Leaky ReLUs allow a small gradient even when a particular neuron is not active. Besides, we include the **\"drop-out\" probability**, as introduced by [8], which defines the probability rate for each neuron to be set to zero at a forward pass to prevent the network from overfitting. However, we explore its effect on the model later in the exercise section of the lab. Initially, we set the dropout probability of each neuron to $p=0.0$ (0%), meaning that none of the neuron activations will be set to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the encoder network\n",
    "class encoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(encoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 384, out 512\n",
    "        self.encoder_L1 = nn.Linear(in_features=ori_subset_transformed.shape[1], out_features=512, bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.encoder_L1.weight) # init weights according to [9]\n",
    "        self.encoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "\n",
    "        # specify layer 2 - in 512, out 256\n",
    "        self.encoder_L2 = nn.Linear(512, 256, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L2.weight)\n",
    "        self.encoder_R2 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 3 - in 256, out 128\n",
    "        self.encoder_L3 = nn.Linear(256, 128, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L3.weight)\n",
    "        self.encoder_R3 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 4 - in 128, out 64\n",
    "        self.encoder_L4 = nn.Linear(128, 64, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L4.weight)\n",
    "        self.encoder_R4 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 5 - in 64, out 32\n",
    "        self.encoder_L5 = nn.Linear(64, 32, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L5.weight)\n",
    "        self.encoder_R5 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 6 - in 32, out 16\n",
    "        self.encoder_L6 = nn.Linear(32, 16, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L6.weight)\n",
    "        self.encoder_R6 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 7 - in 16, out 8\n",
    "        self.encoder_L7 = nn.Linear(16, 8, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L7.weight)\n",
    "        self.encoder_R7 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 8 - in 8, out 4\n",
    "        self.encoder_L8 = nn.Linear(8, 4, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L8.weight)\n",
    "        self.encoder_R8 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 9 - in 4, out 3\n",
    "        self.encoder_L9 = nn.Linear(4, 3, bias=True)\n",
    "        nn.init.xavier_uniform_(self.encoder_L9.weight)\n",
    "        self.encoder_R9 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # init dropout layer with probability p\n",
    "        self.dropout = nn.Dropout(p=0.0, inplace=True)\n",
    "        \n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.encoder_R1(self.dropout(self.encoder_L1(x)))\n",
    "        x = self.encoder_R2(self.dropout(self.encoder_L2(x)))\n",
    "        x = self.encoder_R3(self.dropout(self.encoder_L3(x)))\n",
    "        x = self.encoder_R4(self.dropout(self.encoder_L4(x)))\n",
    "        x = self.encoder_R5(self.dropout(self.encoder_L5(x)))\n",
    "        x = self.encoder_R6(self.dropout(self.encoder_L6(x)))\n",
    "        x = self.encoder_R7(self.dropout(self.encoder_L7(x)))\n",
    "        x = self.encoder_R8(self.dropout(self.encoder_L8(x)))\n",
    "        x = self.encoder_R9(self.encoder_L9(x)) # don't apply dropout to the AE bottleneck\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to instantiate the encoder model. The model will be pushed either on a CPU or available GPU by the execution of the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# intstantiate the encoder network model\n",
    "encoder_train = encoder()\n",
    "\n",
    "# push model to compute device\n",
    "encoder_train = encoder_train.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model is initialised, we can visualise the model structure and review the implemented network architecture by the execution of the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the initialized architectures\n",
    "now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "print('[LOG {}] encoder architecture:\\n\\n{}\\n'.format(now, encoder_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's, now as a next step, complete the `AENN model` implementation by implementing the corresponding decoder network. The decoder also consists of nine fully-connected layers. Furthermore, the decoder network is intended to **symmetrically mirror** the encoder networks architecture by a layer-wise inversion \"3-4-8-16-32-64-128-256-384\" of the encoder network layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the decoder network\n",
    "class decoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(decoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 3, out 4\n",
    "        self.decoder_L1 = nn.Linear(in_features=3, out_features=4, bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.decoder_L1.weight)  # init weights according to [9]\n",
    "        self.decoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "\n",
    "        # specify layer 2 - in 4, out 8\n",
    "        self.decoder_L2 = nn.Linear(4, 8, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L2.weight)\n",
    "        self.decoder_R2 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 3 - in 8, out 16\n",
    "        self.decoder_L3 = nn.Linear(8, 16, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L3.weight)\n",
    "        self.decoder_R3 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 4 - in 16, out 32\n",
    "        self.decoder_L4 = nn.Linear(16, 32, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L4.weight)\n",
    "        self.decoder_R4 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 5 - in 32, out 64\n",
    "        self.decoder_L5 = nn.Linear(32, 64, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L5.weight)\n",
    "        self.decoder_R5 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 6 - in 64, out 128\n",
    "        self.decoder_L6 = nn.Linear(64, 128, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L6.weight)\n",
    "        self.decoder_R6 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "        \n",
    "        # specify layer 7 - in 128, out 256\n",
    "        self.decoder_L7 = nn.Linear(128, 256, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L7.weight)\n",
    "        self.decoder_R7 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 8 - in 256, out 512\n",
    "        self.decoder_L8 = nn.Linear(256, 512, bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L8.weight)\n",
    "        self.decoder_R8 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # specify layer 9 - in 512, out 384\n",
    "        self.decoder_L9 = nn.Linear(in_features=512, out_features=ori_subset_transformed.shape[1], bias=True)\n",
    "        nn.init.xavier_uniform_(self.decoder_L9.weight)\n",
    "        self.decoder_R9 = nn.LeakyReLU(negative_slope=0.4, inplace=True)\n",
    "\n",
    "        # init dropout layer with probability p\n",
    "        self.dropout = nn.Dropout(p=0.0, inplace=True)\n",
    "\n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.decoder_R1(self.decoder_L1(x))\n",
    "        x = self.decoder_R2(self.dropout(self.decoder_L2(x)))\n",
    "        x = self.decoder_R3(self.dropout(self.decoder_L3(x)))\n",
    "        x = self.decoder_R4(self.dropout(self.decoder_L4(x)))\n",
    "        x = self.decoder_R5(self.dropout(self.decoder_L5(x)))\n",
    "        x = self.decoder_R6(self.dropout(self.decoder_L6(x)))\n",
    "        x = self.decoder_R7(self.dropout(self.decoder_L7(x)))\n",
    "        x = self.decoder_R8(self.dropout(self.decoder_L8(x)))\n",
    "        x = self.decoder_R9(self.decoder_L9(x)) # don't apply dropout to the AE output\n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also instantiate the encoder model for CPU or GPU training and convince ourselves that it was successfully initialised by printing and reviewing its architecture by the execution of the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# intstantiate the decoder network model\n",
    "decoder_train = decoder()\n",
    "\n",
    "# push model to compute device\n",
    "decoder_train = decoder_train.to(device)\n",
    "    \n",
    "# print the initialized architectures\n",
    "now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "print('[LOG {}] decoder architecture:\\n\\n{}\\n'.format(now, decoder_train))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like intended? Brilliant! Finally, let's have a look into the number of model parameters that we aim to train in the next steps of the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init the number of model parameters\n",
    "encoder_num_params = 0\n",
    "decoder_num_params = 0\n",
    "\n",
    "# iterate over the distinct encoder parameters\n",
    "for param in encoder_train.parameters():\n",
    "\n",
    "    # collect number of parameters\n",
    "    encoder_num_params += param.numel()\n",
    "    \n",
    "# iterate over the distinct decoder parameters\n",
    "for param in decoder_train.parameters():\n",
    "\n",
    "    # collect number of parameters\n",
    "    decoder_num_params += param.numel()\n",
    "    \n",
    "# print the number of model paramters\n",
    "now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "print('[LOG {}] number of to be trained AENN model parameters: {}.'.format(now, encoder_num_params + decoder_num_params))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, our AENN model encompasses a total number **745'171 model parameters** to be trained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.3. Definition of the Training Loss Function and Learning Rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have implemented the `AENN model` we are ready to train the network. Before starting the training, we need to define an appropriate loss function. Remember, we aim to train our model to learn a set of encoder-decoder model parameters $\\theta$ that minimize the dissimilarity of a given financial transaction $x^{i}$ and its reconstruction $\\hat{x}^{i} = g_\\theta(f_\\theta(x^{i}))$ as faithfully as possible. \n",
    "\n",
    "Thereby, the training objective is to learn a set of optimal shared encoder-decoder model parameters $\\theta^*$ that optimize $\\arg\\min_{\\theta} \\|X - g_\\theta(f_\\theta(X))\\|$ overall transactions or more specifically journal entries $X$. To achieve this optimisation objective, one typically minimises a loss function $\\mathcal{L_{\\theta}}$ as part of the network training. In this lab, we use the **binary-cross-entropy error (BCE)** loss, defined by:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> $\\mathcal{L^{BCE}_{\\theta}}(x^{i};\\hat{x}^{i}) = \\frac{1}{n}\\sum_{i=1}^{n}\\sum_{j=1}^{k} x^{i}_{j} ln(\\hat{x}^{i}_{j}) + (1-x^{i}_{j}) ln(1-\\hat{x}^{i}_{j})$, </center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for a set of $n$-journal entries $x^{i}$, $i=1,...,n$ and their respective reconstructions $\\hat{x}^{i}$ and all journal entry attributes $j=1,...,k$. During training, the BCE loss will penalise models that result in a high dissimilarity between input journal entries and their respective reconstructions. \n",
    "\n",
    "Luckily, an implementation of the BCE loss is already available in PyTorch! It can be instantiated \"off-the-shelf\" via execution of the following PyTorch command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the optimization criterion / loss function\n",
    "loss_function = nn.BCEWithLogitsLoss(reduction='mean')\n",
    "\n",
    "# push the optimization criterion / loss function to compute device \n",
    "loss_function = loss_function.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note: Enabling the parameter `reduction` specifies that the losses are averaged over all observations for each minibatch).\n",
    "\n",
    "Based on the loss magnitude of a certain mini-batch PyTorch automatically computes the gradients. Based on the gradient, the PyTorch library updates and optimises the AENN parameters $\\theta$.\n",
    "\n",
    "We will use the **Adam optimization** as proposed in [11] and set the learning-rate $l = 0.001$. Each mini-batch step the optimizer will update the encoder- and decoder-parameters $\\theta$ values according to degree of reconstruction error. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the learning rate\n",
    "learning_rate = 1e-4\n",
    "\n",
    "#set the paramete optimization strategy of both networks\n",
    "encoder_optimizer = torch.optim.Adam(encoder_train.parameters(), lr=learning_rate)\n",
    "decoder_optimizer = torch.optim.Adam(decoder_train.parameters(), lr=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have successfully implemented and defined the three building blocks of the `AENN model`. Let's take some time to review the `encoder` and `decoder` model definition as well as the `BCE loss`. Please, read the above code and comments carefully and don't hesitate to let us know any questions you might have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Training the Autoencoder Neural Network (AENN) Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will train the deep `AENN model` (as implemented in section 2. of the lab) using the encoded transactional data (created in section 1. of the lab). More specifically, we will have a detailed look into the distinct training steps as well as how to monitor the training progress."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1 Preparing the Network Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have pre-processed the dataset, implemented the AENN and defined the reconstruction error. Let's now start to train a corresponding model for **5 epochs** and a **mini-batch size of 128** journal entries per batch. This training set up implies that the whole dataset will be fed to the AENN 5 times in chunks of 128 journal entries yielding to **4,165 mini-batches** (533,009 journal entries / 128 journal entries per mini-batch) per epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify training parameters\n",
    "num_epochs = 5\n",
    "mini_batch_size = 128"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the training phase, we will fetch the individual mini-batches of the entire population of journal entries. To achieve this, we will use PyTorch's `DataLoader` that provides single- or multi-process iterators over a given dataset to load one mini-batch at a time. By enabling `shuffle=True`, the data will be shuffled at every epoch before feeding it to the network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert pre-processed transactional data to PyTorch tensor\n",
    "torch_dataset = torch.from_numpy(ori_subset_transformed.values).float()\n",
    "\n",
    "# push pre-processed transactional data to compute device\n",
    "torch_dataset = torch_dataset.to(device)\n",
    "\n",
    "# convert to pytorch tensor - none cuda enabled\n",
    "dl = dataloader.DataLoader(torch_dataset, batch_size=mini_batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2 Running the Network Training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we start training the model. The training procedure of each mini-batch is performed as follows: \n",
    "\n",
    ">1. do a forward pass through the encoder-decoder part, \n",
    ">2. compute the binary-cross-entropy reconstruction loss $\\mathcal{L^{BCE}_{\\theta}}(x^{i};\\hat{x}^{i})$, \n",
    ">3. do a backward pass through the encoder-decoder part, and \n",
    ">4. update the parameters of the encoder $f_\\theta(\\cdot)$ and decoder $g_\\theta(\\cdot)$ networks.\n",
    "\n",
    "To ensure learning while training our `AENN model,` we will monitor whether the loss decreases with progressing training. Therefore, we obtain and evaluate the reconstruction performance of the entire dataset after each training epoch. Based on this evaluation, we can conclude the training progress and whether the loss is converging (indicating that the model might not improve any further).\n",
    "\n",
    "The following elements of the network training code below should be given particular attention:\n",
    " \n",
    ">- `reconstruction_loss.backward()` computes the gradients based on the magnitude of the reconstruction loss,\n",
    ">- `encoder_optimizer.step()` and `decoder_optimizer.step()` updates the network parameters based on the gradient.\n",
    "\n",
    "Also, after each training epoch, we want to save a checkpoint for both the actual `encoder` and `decoder` model. The saved model checkpoints contain a snapshot of the trained model parameter values upon completion of a training epoch. In general, it is good practice to keep checkpoints at regular intervals during training. In case your system crashes during training, you can continue from the last checkpoint rather than start over from scratch.\n",
    "\n",
    ">- `torch.save()`: saves a checkpoint of the actual encoder and decoder model parameter values to disc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init collection of training epoch losses\n",
    "train_epoch_losses = []\n",
    "\n",
    "# set the model in training mode (apply dropout when needed)\n",
    "encoder_train.train()\n",
    "decoder_train.train()\n",
    "\n",
    "# init the best loss\n",
    "best_loss = np.inf\n",
    "\n",
    "# train autoencoder model\n",
    "for epoch in range(num_epochs + 1):\n",
    "\n",
    "    # init collection of epoch losses\n",
    "    train_mini_batch_losses = []\n",
    "    \n",
    "    # init mini batch counter\n",
    "    mini_batch_count = 0\n",
    "        \n",
    "    # iterate over all mini-batches\n",
    "    for mini_batch_data in dl:\n",
    "\n",
    "        # increase mini batch counter\n",
    "        mini_batch_count += 1\n",
    "\n",
    "        # push mini batch data to compute device\n",
    "        mini_batch_data = mini_batch_data.to(device)\n",
    "\n",
    "        # =================== (1) forward pass ===================================\n",
    "\n",
    "        # run forward pass\n",
    "        z_representation = encoder_train(mini_batch_data) # encode mini-batch data\n",
    "        mini_batch_reconstruction = decoder_train(z_representation) # decode mini-batch data\n",
    "        \n",
    "        # =================== (2) compute reconstruction loss ====================\n",
    "\n",
    "        # determine reconstruction loss\n",
    "        reconstruction_loss = loss_function(mini_batch_reconstruction, mini_batch_data)\n",
    "        \n",
    "        # =================== (3) backward pass ==================================\n",
    "\n",
    "        # reset graph gradients\n",
    "        decoder_optimizer.zero_grad()\n",
    "        encoder_optimizer.zero_grad()\n",
    "\n",
    "        # run backward pass\n",
    "        reconstruction_loss.backward()\n",
    "        \n",
    "        # =================== (4) update model parameters ========================\n",
    "\n",
    "        # update network parameters\n",
    "        decoder_optimizer.step()\n",
    "        encoder_optimizer.step()\n",
    "\n",
    "        # =================== monitor training progress ==========================\n",
    "\n",
    "        # print training progress each 1.000 mini-batches\n",
    "        if mini_batch_count % 1000 == 0:\n",
    "            \n",
    "            # print mini batch reconstuction results\n",
    "            now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "            print('[LOG {}] epoch: [{}/{}], batch: {}, batch-train-loss: {}'.format(str(now), str(epoch+1), str(num_epochs), str(mini_batch_count), str(np.round(reconstruction_loss.item(), 8))))\n",
    "            \n",
    "        # collect mini-batch loss\n",
    "        train_mini_batch_losses.extend([reconstruction_loss.item()])\n",
    "\n",
    "    # =================== evaluate model performance =============================\n",
    "    \n",
    "    # determine mean min-batch loss of epoch\n",
    "    train_epoch_loss = np.mean(train_mini_batch_losses)\n",
    "                                 \n",
    "    # print training epoch results\n",
    "    now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "    print('[LOG {}] epoch: [{}/{}], epoch-train-loss: {}'.format(str(now), str(epoch+1), str(num_epochs), str(np.round(train_epoch_loss, 8))))\n",
    "\n",
    "    # determine mean min-batch loss of epoch\n",
    "    train_epoch_losses.append(train_epoch_loss)\n",
    "    \n",
    "    # =================== save model snapshot to disk ============================\n",
    "    \n",
    "    # case: new best model trained\n",
    "    if train_epoch_loss < best_loss:\n",
    "    \n",
    "        # save trained encoder model file to disk\n",
    "        encoder_model_name = \"ep_{}_encoder_model.pth\".format((epoch+1))\n",
    "        torch.save(encoder_train.state_dict(), os.path.join(\"./models\", encoder_model_name))\n",
    "\n",
    "        # save trained decoder model file to disk\n",
    "        decoder_model_name = \"ep_{}_decoder_model.pth\".format((epoch+1))\n",
    "        torch.save(decoder_train.state_dict(), os.path.join(\"./models\", decoder_model_name))\n",
    "        \n",
    "        # update best loss\n",
    "        best_loss = train_epoch_loss\n",
    "\n",
    "        # print epoch loss\n",
    "        now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "        print('[LOG {}] epoch: [{}/{}], new best epoch-train-loss: {} found'.format(str(now), str(epoch+1), str(num_epochs), str(np.round(train_epoch_loss, 8))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now visualise the loss magnitude obtained for each training epoch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# add grid\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "# plot the training epochs vs. the epochs' prediction error\n",
    "ax.plot(np.array(range(1, len(train_epoch_losses)+1)), train_epoch_losses, label='epoch loss (blue)')\n",
    "\n",
    "# add axis legends\n",
    "ax.set_xlabel(\"[Training Epoch $e_i$]\", fontsize=10)\n",
    "ax.set_ylabel(\"[Reconstruction Error $\\mathcal{L}^{BCE}$]\", fontsize=10)\n",
    "\n",
    "# set plot legend\n",
    "plt.legend(loc=\"upper right\", numpoints=1, fancybox=True)\n",
    "\n",
    "# add plot title\n",
    "plt.title('Training Epochs $e_i$ vs. Reconstruction Error $L^{BCE}$', fontsize=10);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be observed that, after training the `AENN model` for five training epochs, the reconstruction error starts to decrease. This observation indicates that the model already did an excellent job in learning the distinct attribute pattern of the journal entries. However, judging from the loss visualisation above it can also conclude that, the model could probably be trained a couple more epochs as the reconstruction error continues to decrease."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Evaluating the Model's Anomaly Detection Capability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will evaluate the anomaly detection capability of the (pre-trained) `AENN model`. The comparison will encompass the **local** as well as the **global anomalies** contained in the dataset of journal entries. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1 Loading a (pre-trained) Model Checkpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before starting our evaluation, let's load the best performing `AENN model` or an already pre-trained model (as done below). Remember, that we stored a snapshot of the model parameters after each training epoch to our local model directory. We will now load one of the pre-trained checkpoints that was trained for **50 training epochs**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# restore pretrained model checkpoint\n",
    "encoder_model_name = 'https://raw.githubusercontent.com/financial-data-science/CFDS/master/lab_11/models/ep_50_encoder_model.pth'\n",
    "decoder_model_name = 'https://raw.githubusercontent.com/financial-data-science/CFDS/master/lab_11/models/ep_50_decoder_model.pth'\n",
    "\n",
    "# read stored model from the remote location\n",
    "encoder_bytes = urllib.request.urlopen(encoder_model_name)\n",
    "decoder_bytes = urllib.request.urlopen(decoder_model_name)\n",
    "\n",
    "# load tensor from io.BytesIO object\n",
    "encoder_buffer = io.BytesIO(encoder_bytes.read())\n",
    "decoder_buffer = io.BytesIO(decoder_bytes.read())\n",
    "\n",
    "# init evaluation encoder and decoder model\n",
    "encoder_eval = encoder()\n",
    "decoder_eval = decoder()\n",
    "\n",
    "# push encoder and decoder model to compute device\n",
    "encoder_eval = encoder_eval.to(device)\n",
    "decoder_eval = decoder_eval.to(device)\n",
    "\n",
    "# load trained models\n",
    "encoder_eval.load_state_dict(torch.load(encoder_buffer))\n",
    "decoder_eval.load_state_dict(torch.load(decoder_buffer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2 Evaluation of the (pre-trained) Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once obtained, let's use the model to reconstruct the individual journal entries of the entire dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set networks in evaluation mode (don't apply dropout)\n",
    "encoder_eval.eval()\n",
    "decoder_eval.eval()\n",
    "\n",
    "# reconstruct encoded transactional data\n",
    "reconstruction = decoder_eval(encoder_eval(torch_dataset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, great that went smoothly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.3 Assessment of the Individual Journal Entries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's assess the reconstruction quality of individual journal entries of the dataset. To achieve this, we collect the reconstruction errors of each journal entry by executing the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# init binary cross entropy errors\n",
    "reconstruction_loss_transaction = np.zeros(reconstruction.size()[0])\n",
    "\n",
    "# iterate over all detailed reconstructions\n",
    "for i in range(0, reconstruction.size()[0]):\n",
    "\n",
    "    # determine reconstruction loss - individual transactions\n",
    "    reconstruction_loss_transaction[i] = loss_function(reconstruction[i], torch_dataset[i]).item()\n",
    "\n",
    "    if(i % 100000 == 0):\n",
    "\n",
    "        ### print conversion summary\n",
    "        now = dt.datetime.utcnow().strftime(\"%Y.%m.%d-%H:%M:%S\")\n",
    "        print('[LOG {}] collected individual reconstruction loss of: {:06}/{:06} transactions'.format(now, i, reconstruction.size()[0]))\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have collected individual reconstruction errors, let's visualise them accordingly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare plot\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# assign unique id to transactions\n",
    "plot_data = np.column_stack((np.arange(len(reconstruction_loss_transaction)), reconstruction_loss_transaction))\n",
    "\n",
    "# obtain regular transactions as well as global and local anomalies\n",
    "regular_data = plot_data[label == 'regular']\n",
    "global_outliers = plot_data[label == 'global']\n",
    "local_outliers = plot_data[label == 'local']\n",
    "\n",
    "# plot reconstruction error scatter plot\n",
    "ax.scatter(regular_data[:, 0], regular_data[:, 1], c='C0', alpha=0.4, marker=\"o\", label='regular') # plot regular transactions\n",
    "ax.scatter(global_outliers[:, 0], global_outliers[:, 1], c='C1', marker=\"^\", label='global') # plot global outliers\n",
    "ax.scatter(local_outliers[:, 0], local_outliers[:, 1], c='C2', marker=\"^\", label='local') # plot local outliers\n",
    "\n",
    "# add plot legend of transaction classes\n",
    "ax.legend(loc='best');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The visualisation reveals that the pre-trained model can reconstruct the majority of regular journal entries while failing to do so, for the anomalous ones. As a result, the model reconstruction error can be used to distinguish both **\"global\" anomalies (green)** and **\"local\" anomalies (red)** from the regular journal entries (blue).\n",
    "\n",
    "To further investigate our observation and confirm the initial assumption, let's have a closer look into the journal entries exhibiting a **binary cross-entropy reconstruction error >= 0.1**. We assume that these journal entries correspond to the \"global\" anomalies of the original dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# append labels to original dataset\n",
    "ori_dataset['label'] = label\n",
    "\n",
    "# inspect transactions exhibiting a reconstruction error >= 0.1\n",
    "ori_dataset[reconstruction_loss_transaction >= 0.1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now also have a closer look into the journal entries exhibiting a **binary cross-entropy reconstruction error >= 0.02 and < 0.1**. We assume that these journal entries mostly correspond to the \"local\" anomalies of the original dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inspect transactions exhibiting a reconstruction error < 0.1 and >= 0.02\n",
    "ori_dataset[(reconstruction_loss_transaction >= 0.02) & (reconstruction_loss_transaction < 0.1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Evaluating the Model's Latent Space Representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a real-world scenario, it is usually also beneficial to inspect the latent space representations of each individual journal entry that was learned by the `AENN model`. This to determine if the autoencoder learned a meaningful representation and to obtain an impression of the structural characteristics evident in the data. To achieve this, we will propagate the journal entries through the encoder network of the trained model to obtain the respective **latent space representation** for each entry (also referred to as \"embedding\", (the activation pattern of each journal entry at the bottleneck neurons)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now load one, even longer pre-trained checkpoint, that was trained for **100 training epochs**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# restore pretrained model checkpoint\n",
    "encoder_model_name = 'https://raw.githubusercontent.com/financial-data-science/CFDS/master/lab_11/models/ep_100_encoder_model.pth'\n",
    "decoder_model_name = 'https://raw.githubusercontent.com/financial-data-science/CFDS/master/lab_11/models/ep_100_decoder_model.pth'\n",
    "\n",
    "# read stored model from the remote location\n",
    "encoder_bytes = urllib.request.urlopen(encoder_model_name)\n",
    "decoder_bytes = urllib.request.urlopen(decoder_model_name)\n",
    "\n",
    "# load tensor from io.BytesIO object\n",
    "encoder_buffer = io.BytesIO(encoder_bytes.read())\n",
    "decoder_buffer = io.BytesIO(decoder_bytes.read())\n",
    "\n",
    "# init evaluation encoder and decoder model\n",
    "encoder_eval = encoder()\n",
    "decoder_eval = decoder()\n",
    "\n",
    "# push encoder and decoder model to compute device\n",
    "encoder_eval = encoder_eval.to(device)\n",
    "decoder_eval = decoder_eval.to(device)\n",
    "\n",
    "# load trained models\n",
    "encoder_eval.load_state_dict(torch.load(encoder_buffer))\n",
    "decoder_eval.load_state_dict(torch.load(decoder_buffer))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's obtain the distinct representations of the journal entries and convert them to Pandas `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z_representation = pd.DataFrame(encoder_eval(torch_dataset).data.numpy(), columns=['z1', 'z2', 'z3'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Furthermore, let's add the original labels of each journal entry to the obtained representations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z_representation['label'] = label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the first ten obtained embeddings and corresponding labels: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "z_representation.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When designing the AENN model architecture, we intentionally chose a bottleneck layer consisting of **3 neurons**. This provides us with the ability to visualise the distinctive activation pattern using the 3d plotting capabilities of Python's `Matplotlib` library.\n",
    "\n",
    "Let's draw an interactive 3d scatter plot of the latent space representations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "from importlib import reload\n",
    "reload(plt)\n",
    "\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "# init the plot\n",
    "fig = plt.figure(figsize=(8, 8))\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# init 3D plotting\n",
    "ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=30, azim=120)\n",
    "\n",
    "# set axis paramaters of subplot\n",
    "ax.grid(linestyle='dotted')\n",
    "\n",
    "\n",
    "# plot regular transactions\n",
    "regular = z_representation[z_representation['label'] == 'regular']\n",
    "ax.scatter(regular['z1'], regular['z2'], regular['z3'], c='C0', alpha=0.4, marker=\"o\")\n",
    "\n",
    "# plot first order anomalous transactions\n",
    "global_anomalies = z_representation[z_representation['label'] == 'global']\n",
    "ax.scatter(global_anomalies['z1'], global_anomalies['z2'], global_anomalies['z3'], c='C1', s=100, marker=\"^\")\n",
    "\n",
    "# plot second order anomalous transactions\n",
    "local_anomalies = z_representation[z_representation['label'] == 'local']\n",
    "ax.scatter(local_anomalies['z1'], local_anomalies['z2'], local_anomalies['z3'], c='C2', s=100, marker=\"^\");\n",
    "\n",
    "# set axis labels\n",
    "ax.set_xlabel('activation [$z_1$]', weight='normal', fontsize=12)\n",
    "ax.set_ylabel('activation [$z_2$]', weight='normal', fontsize=12)\n",
    "ax.set_zlabel('activation [$z_3$]', weight='normal', fontsize=12)\n",
    "\n",
    "# set plot title\n",
    "plt.title('Learned Latent Space Representations', fontsize=20)\n",
    "\n",
    "# show the 3-dimensional plot\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We recommend you to try the following exercises as part of the lab:\n",
    "\n",
    "**1. Add of the two additional journal entry attributes `WAERS` and `BUKRS`.**\n",
    "\n",
    ">Pre-process the journal entry data and learn an `AENN model` including also the attributes `WAERS` and `BUKRS`. Therefore, (1) plot and inspect the distribution of the distinct values observable for both attributes, (2) encode the values of the attributes using the `get_dummies()` method provided by the Pandas library, and (3) merge your encoding results with the `ori_subset_transformed` data frame (upon successful merge the one-hot encoded dataset should encompass a total dimensionality of 638 instead of 384 columns). Ultimately, train an `AENN model` including both attributes and evaluate its anomaly detection performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ***************************************************\n",
    "# INSERT YOUR CODE HERE\n",
    "# ***************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Apply a `dropout` throughout the network training.**\n",
    "\n",
    ">Set the `dropout` probability to `0.2` (20%) and re-start the training procedure. What impact do you observe in terms of training performance / reconstruction loss? Please, note that after changing the `dropout` probability of the encoder and decoder model you need to execute all successive cells starting from section 2.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ***************************************************\n",
    "# INSERT YOUR CODE HERE\n",
    "# ***************************************************"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Train and evaluate a `shallow` autoencoder model.**\n",
    "\n",
    "> The lab model architecture resulted in a good anomaly detection accuracy. Let's see how the reconstruction performance change if **several of the hidden layers** will be removed. First, adjust the encoder and decoder model definitions in section 2.2 accordingly (you may want to use the code snippets shown below). Then, follow all the instructions for training from scratch. Please note that after altering the encoder and decoder network architecture in section 2.2, you need to execute all successive cells starting from section 2.2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the shallow encoder network \n",
    "# containing only a single layer\n",
    "class shallow_encoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(encoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 618, out 3\n",
    "        self.encoder_L1 = nn.Linear(in_features=ori_subset_transformed.shape[1], out_features=3, bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.encoder_L1.weight) # init weights according to [9]\n",
    "        self.encoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "        \n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.encoder_R1(self.encoder_L1(x)) # don't apply dropout to the AE bottleneck\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# implementation of the shallow decoder network \n",
    "# containing only a single layer\n",
    "class shallow_decoder(nn.Module):\n",
    "\n",
    "    def __init__(self):\n",
    "\n",
    "        super(decoder, self).__init__()\n",
    "\n",
    "        # specify layer 1 - in 3, out 618\n",
    "        self.decoder_L1 = nn.Linear(in_features=3, out_features=ori_subset_transformed.shape[1], bias=True) # add linearity \n",
    "        nn.init.xavier_uniform_(self.decoder_L1.weight)  # init weights according to [9]\n",
    "        self.decoder_R1 = nn.LeakyReLU(negative_slope=0.4, inplace=True) # add non-linearity according to [10]\n",
    "\n",
    "    def forward(self, x):\n",
    "\n",
    "        # define forward pass through the network\n",
    "        x = self.decoder_R1(self.decoder_L1(x)) # don't apply dropout to the AE output\n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lab Summary:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this eleventh lab, a step by step introduction into **design, implementation, training and evaluation** of an autoencoder neural network-based anomaly detection approach is presented. The trained autoencoder neural network's reconstruction error obtainable for each journal entry can be interpreted as a highly adaptive anomaly assessment. The code provided in this lab offers a blueprint for the development and testing of more complex anomaly detection strategies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may want to execute the content of your lab outside of the Jupyter notebook environment, e.g. on a compute node or a server. The cell below converts the lab notebook into a standalone and executable python script. Pls. note that to convert the notebook, you need to install Python's **nbconvert** library and its extensions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# installing the nbconvert library\n",
    "!pip3 install nbconvert\n",
    "!pip3 install jupyter_contrib_nbextensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now convert the Jupyter notebook into a plain Python script:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!jupyter nbconvert --to script cfds_colab_11.ipynb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Major elements of the lab content are inspired by the publication \"Detection of Anomalies in Large Scale Accounting Data using Deep Autoencoder Networks\", of M. Schreyer, T. Sattarov, D. S. Borth, A. Dengel, and B. Reimer, 2017 (arXiv preprint available under: https://arxiv.org/abs/1709.05254).\n",
    "\n",
    "[1] ACFE, \"Report to the Nations on Occupational Fraud and Abuse\", The 2016 Global Fraud Study, Association of Certified Fraud Examiners (ACFE), 2016.\n",
    "\n",
    "[2] J. T. Wells, \"Corporate Fraud Handbook: Prevention and Detection\", John Wiley & Sons, 2017.\n",
    "\n",
    "[3] PwC, \"Pulling Fraud Out of the Shadows\", The Global Economic Crime and Fraud Survey 2018, PricewaterhouseCoopers LLP, 2018.\n",
    "\n",
    "[4] S. Markovitch, P. Willmott, \"Accelerating the digitization of business processes\", McKinsey & Company (2014) 1–5.\n",
    "\n",
    "[5] SAP, SAP Global Corporate Affairs, Corporate Factsheet 2017, 2017.\n",
    "\n",
    "[6] E. A. Lopez-Rojas , A. Elmir, and S. Axelsson, \"PaySim: A financial mobile money simulator for fraud detection\", In: The 28th European Modeling and Simulation Symposium-EMSS, Larnaca, Cyprus, 2016.\n",
    "\n",
    "[7] G. E. Hinton, and R. R. Salakhutdinov, \"Reducing the dimensionality of data with neural networks\", science 313, no. 5786: 504-507, 2006.\n",
    "\n",
    "[8] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, \"Dropout: A simple way to prevent neural networks from overfitting\", The Journal of Machine Learning Research, 15(1), 1929-1958, 2014.\n",
    "\n",
    "[9] X. Glorot and Y. Bengio, \"Understanding the difficulty of training deep feedforward neural networks\", Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS), 9:249–256, 2010.\n",
    "\n",
    "[10] B. Xu, N. Wang, T. Chen, and M. Li, \"Empirical Evaluation of Rectified Activations in Convolution Network\", ICML Deep Learning Workshop, pages 1–5, 2015.\n",
    "\n",
    "[11] D. P. Kingma and J. Ba, \"Adam: A method for stochastic optimization\", International Conference on Learning Representations (ICLR). 2015.\n",
    "\n",
    "[12] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R. R. Salakhutdinov. \"Improving neural networks by preventing co-adaptation of feature detectors\", Technical Report, 2012.\n",
    "\n",
    "[13] D. P. Kingma, M. Welling. \"Auto-encoding variational bayes\", arXiv preprint arXiv:1312.6114, 2013.\n",
    "\n",
    "[14] Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., & Frey, B., \"Adversarial autoencoders\", arXiv preprint arXiv:1511.05644, 2015.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
